Tutorial ML 1

A common task in BCI reasearch is to test a machine learning model (MLM) on a large amount of real data. This tutorial uses the FII-BCI corpus in NY format as an example.

The tutorial shows how to

Select databases and sessions from the FII-BCI corpus accoording to:
- BCI Paradigm (Motor Imagery or P300)
- availability of specific classes
- minimum number of trials per class
Run a cross-validation for all selected sessions in all selected databases and show a summary of the cross-validation results for each session.

Info

As a MLM, the MDM Riemannian classifier employing the affine-invariant (Fisher-Rao) metric is used (Barachant et al., 2012), (Congedo et al., 2017). As a covariance matrix estimator, the linear shrinkage estimator of (Ledoit and Wolf, 2004) is used. These are state-of-the art settings used as default in Eegle.

For each session, an 8-fold stratified cross-validation is run. The summary of results comprises the mean and standard deviation of the balanced accuracy obtained across the folds as well as the z-score and p-value of the cross-validation test-statistic.

Tell julia you want to use the Eegle package

using Eegle

Create a function to print the results. The function takes as arguments the serial number of the file in the database and the result of the cross-validation, which is a CVres structure.

pr(f, res) = println("File ", f, ". mean(sd): ", round(res.avgAcc*100, digits=2),
                        "% (± ", round(res.stdAcc*100, digits=2), " %); ", 
                        "z(p-value): ", round(res.z, digits=4), "(", res.p, ")")

Perform the cross-validation on all available MI databases featuring the left_hand and right_hand class (see selectDB):

MIDir = joinpath(homedir(), "FII corpus", "NY","MI") # path to MI databases
classes = ["feet", "right_hand"]
DBs = selectDB(MIDir, :MI; classes);

for (db, DB) ∈ enumerate(DBs)
    println("\nDatabase: ", DB.dbName)
    println("\nmean and sd balanced accuracy, z and p-value against chance level:")
    for (f, file) ∈ enumerate(DB.files)
        pr(f, crval(file; bandPass=(8, 32), classes))
    end
end

Perform the cross-validation on all available P300 databases featuring at least 25 trials for both the target and non-target classes. For P300 there is no need to specify these two classes as they are the default:

P300Dir = joinpath(homedir(), "FII corpus","NY","P300") # path to P300 databases
DBs = selectDB(P300Dir, :P300; minTrials = 25);

for (db, DB) ∈ enumerate(DBs)
    println("\nDatabase: ", DB.dbName)
    println("\nmean and sd balanced accuracy, z and p-value against chance level:")
    for (f, file) ∈ enumerate(DB.files)
        pr(f, crval(file; bandPass=(1, 24)))
    end
end

For all possible options in running cross-validations, see crval.

Tip

Do not use Julia's @threaded macro in the internal loops above as function crval is already multi-threaded across folds.